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Technical Appendix 

This appendix describes the analytical procedures used to project the status of 
schools in the Great Lakes state toward meeting AYP requirements from 2005 to 2014. 

Meeting AYP Targets 

To “make AYP,” the school as a whole and each numerically significant subgroup 
must meet “Annual Measurable Objectives” (annual objectives). States set their own 
annual objectives and these requirements increase over time toward the goal of 100% of 
students reaching proficiency in 2014. Reaching proficiency is measured by students 
meeting standards on the state assessment. In addition, each state sets the minimum 
student count necessary for a student subgroup to reach numerical significance. Once a 
subgroup reaches the minimum student count threshold, it is included in the AYP 



calculations. 




The observed percentage of student proficiency for each subject area, overall 
school, and numerically significant subgroup combination represents a statistical estimate 
of the “true” proportion proficient within a given grouping. Statistical uncertainty, then, 
plays a role in the likelihood that an observed proficiency rate exceeds the corresponding 
annual objective. Rogosa 1 makes this point in describing his study quantifying the 
effects of statistical uncertainty associated with NCLB: 

Statistical properties of school and subgroup scores are important in 
understanding accountability systems... statistical variability in the school and 
subgroup scores makes growth targets far more formidable than they might 
appear, in large part because of the subgroup requirements (as each of the 
subgroups has larger uncertainty than the school index). 

This study draws from Rogosa’ s work. True proportions proficient are projected 
under a variety of scenarios for the whole school and for each numerically significant 
subgroup. In practice, these true proficiency rates cannot be observed - they can only be 
estimated (subject to statistical variability mentioned above). The likelihood of observing 
proficiency greater than the associated annual objective is calculated based on group size 
and projected true proficiency. Aggregating these likelihoods over all group 
measurements (the school as a whole and for each subgroup) and over all subjects 
(mathematics and English/language arts) gives an overall likelihood of a school making 
AYP. Furthermore, aggregating these likelihoods over all schools in a given year 
provides an estimate of statewide rates of school “success” and “failure” relative to AYP. 

Consider the following example 2 : School A consists of 200 students, 100 percent 
of whom participate in both the English/language arts (EL A) assessment and the 
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mathematics assessment. School A is a “perfectly homogenous” school - all the students 



belong to the same ethnic subgroup, and no students are designated Limited English 
Proficiency (hereafter “LEP”), Socioeconomically disadvantaged (“SD”), or disabled 
(“SWD”). This school has only one numerically significant subgroup (the whole school) 
and must meet two annual objective targets - one for ELA and one for math. 

Assume the annual objectives which School A must meet to make AYP are 40 
percent of students meeting the standards on the state test for both ELA and mathematics. 
Put another way, School A must have at least 80 students (40 percent of its 200 total 
students) proficient on both its ELA and mathematics assessments to make AYP. 

Also assume that School A has a “true” proportion proficient of 45 percent in both 
ELA and mathematics. If we were able to observe this true proficiency without error, we 
would clearly designate School A as having made AYP because it exceeds the 40 percent 
requirement in ELA and math. This true proficiency cannot be observed, however; it is 
subject to statistical uncertainty in the form of sampling variability. 

What, then, is the likelihood that the observed proficiency exceeds the 40 percent 
requirement or 80 out of 200 students meeting the standards? Probability calculations 
based on the binomial distribution provide an estimate of the likelihood that this 80- 
student requirement will be met given a true proficiency level of 45 percent. As Rogosa 4 
notes, not only can calculations based on the binomial distribution provide an estimate of 
likelihood of school success relative to AYP, but citing CCSSO 5 he also states, “even 
more crude large-sample normal theory approximations to binomial variability are at the 
core of the state NCLB plans that use the (unfortunate) confidence interval adjustments to 
the AMO.” 
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Figure 1 details the binomial calculation described above. School A is most likely 
to have 90 students observed proficient - no surprise since its true proficiency is 45 
percent. Given this true proficiency, School A has a 91 percent chance of meeting its 
target of 80 proficient students - a high percentage, but by no means a certainty. 



Figure 1 



EXAMPLE 1: LIKELIHOOD THAT 
SCHOOL A MEETS ELA AMO 




70 72 74 76 78 80 82 84 



88 90 92 94 96 98 100 102 104 106 108 110 



# Students Observed Proficient 



School A likelihood of meeting AMO = 91.2% 

To see the effect of having multiple subgroups, consider a second example. 

School B consists of 200 students, 100 percent of whom participate in both the 
English/language arts (ELA) assessment and the mathematics assessment. In contrast to 
School A, students in School B are perfectly split between two numerically significant 
student subgroups, each with 100 students. Similar to School A, no students are 
designated as LEP, SD, or SWD. School B must meet four annual objective targets - one 
for each of the two subgroups for both ELA and math. 
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Similar to School A, assume that each subgroup of School B has a “true” 
proportion proficient of 45 percent in both ELA and mathematics. Again, if we were able 
to observe this true proficiency without error, School B makes AYP because the 
percentage of proficient students for each subgroup exceeds the 40 percent target in both 
ELA and math. This true proficiency cannot be observed, however, so the task at hand is 
to estimate the likelihood that the observed proficiency exceeds the 40 percent target. 

Consider first the likelihood that Subgroup 1 meets the 40 percent ELA target. At 
least 40 students (of the 100 in the subgroup) must be observed proficient in order to 
meet this target. As detailed in Figure 2, given a true proficiency level of 45 percent, 
School B’s Subgroup 1 has about an 82 percent chance of meeting its target of 40 
proficient students. 

Figure 2 



LIKELIHOOD THAT SCHOOL B, 
SUBGROUP 1 MEETS ELA AMO 



P[X < 40] 
= 0.183 
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P[X> 40] = 0.817 




20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 



X = # Students Observed Proficient 



Subgroup 1 likelihood of meeting AMO = 81.7% 
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Because the two ethnic subgroups do not overlap, the likelihood that the school 



meets both ELA requirements (one for each subgroup) is the joint likelihood that both 
subgroups meet their individual ELA annual objectives. The likelihood that subgroup 1 
meets its targets is 81.7 percent, detailed above. Subgroup 2 has the same true 
proficiency as subgroup 1 (45 percent) and is of identical size; thus the likelihood that 
subgroup 2 meets the 40 percent target is also 81.7 percent. Because the groups are 
independent (non-overlapping), the joint likelihood that they meet their targets is the 
product of the independent likelihoods (0.817 * 0.817 = 66.7 percent). 

Through this example, the effect of multiple subgroups becomes evident. Given 
the same annual objectives and true proficiency assumptions, School A composed of a 
homogenous, 200-student population, has about a 91 percent chance of making AYP in 
ELA. School B has less than 67 percent chance to make AYP. 

The examples provide likelihoods of making AYP targets in ELA only. Schools 
must also meet similar requirements in mathematics in order to make AYP. Rogosa 6 
provided an approximation for the joint likelihood of meeting both ELA and mathematics 
targets, noting that this likelihood should fall somewhere between the “single test 
probability” and the probabilities under independent trials. This study uses a similar 
approximation, using the geometric mean of (a) the product of the likelihoods of meeting 
the two targets (i.e., independent trials), and (b) the geometric mean of the two 
likelihoods (to approximate the “single test probability”). 

Applying this approximation to the example schools yields an 87.1 percent 
likelihood of School A meeting both math and ELA targets, and a 54.5 percent likelihood 
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that School B meets both targets. Schools with higher numbers of smaller-sized 



subgroups face greater challenges making AYP based on this characteristic alone. 

Many states have employed a variety of methodological strategies to increase the 
likelihood that schools and subgroups meet their AYP targets such as, “confidence 
intervals” (IL, IN, MN, and WI), rolling averages (OH), and awarding partial credit for 
nearly-proficient students (MN). As part of projecting the outcomes for schools in each 
state, our analyses take into account all the strategies employed in the AYP parameters 

7 

for each state accountability system. 

Meeting Safe Harbor 

Schools may also demonstrate that they have made adequate yearly growth 
through another option known as “Safe Harbor.” The Safe Harbor criteria include, (1) 
the school or subgroup must reduce the percentage of students not meeting standards by 
at least ten percent and (2) the school or subgroup must also demonstrate adequate yearly 
progress on an alternative criterion, such as attendance rate or graduation rate. 

This study includes projections of schools eligible for Safe Harbor under the first 
criterion; no attempt is made to project growth against the second criterion. It must also 
be noted that the schools eligible for Safe Harbor have not necessarily made AYP. 
Therefore, the projections in this study provide an upper bound, or a conservative 
estimate, of the schools that will actually achieve Safe Harbor; in reality, some proportion 
of schools eligible will fail to demonstrate progress on the alternative measure and will 
subsequently be designated as failing to meet AYP. 
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Eligibility for Safe Harbor is estimated using the same strategy that is used for 
AYP, but it is based on the Safe Harbor targets. Given a “true” proportion proficiency 
within a given subgroup or school, what is the l ik elihood of observing a decrease of at 
least 10 percent in the proportion non-proficient students relative to the previous year? 
Safe Harbor requires growth relative to a school or subgroup’s own proficiency rather 
than absolute annual objectives. Thus, Safe Harbor eligibility behaves differently than 
making AYP based on the annual objectives. In fact, the annual objectives may actually 
be easier to meet than their Safe Harbor in some cases. For illustrative purposes, we will 
consider a school that has 50 percent of students proficient in mathematics in 2006. In 
2007, the Mathematics annual objective is 53 percent. Meeting the 2007 annual objective 
requires this school to increase Mathematics proficiency by three percent or greater from 
2006 to 2007. Safe Harbor, on the other hand, requires a decrease of 10% or greater in 
the number of non-proficient students. Safe Harbor requires that this decrease by at least 
10 percent - to no greater than 45 percent non-proficient, corresponding to proficiency of 
55 percent proficient or greater. In order to trigger the Safe Harbor option, the 
percentage of students meeting proficiency must reach 55 percent or greater. In this case, 
the 2007 Safe Harbor target of 55 percent exceeds the 2007 annual objective of 53 
percent. The example school is more likely to achieve the annual objective than to meet 
the Safe Harbor criteria. 

The reader of the main report may be initially surprised to see that, in some cases, 
eligibility for Safe Harbor is actually greater under less aggressive growth scenarios. For 
example, 25.4 percent of Wisconsin schools are Safe Harbor eligible in 2014 under the 
Medium Growth scenario compared to 40.7 percent of Wisconsin schools eligible under 
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the Low Growth scenario. The reason is because Safe Harbor is based on relative targets. 
AYP, by contrast, is based on an absolute standard and the relationship between the 
school projections, and AYP status is more direct. As the projected growth increases, 
success relative to AYP increases monotonically as well. 

Precisely because Safe Harbor is based relative targets, one could argue that it 
violates the goals of NCLB. For demonstration purposes, Figure 1 provides an analysis 
of how Safe Harbor allows those groups that are currently least proficient to remain less 
proficient by a significant margin in 2014. As Figure 3 shows, today’s least proficient 
subgroups could actually meet AYP every single year yet still have roughly 33 percent of 
students fail to meet standards. 



Figure 1 



SAFE HARBOR TARGETS BY STARTING POINT 



Proficiency Required by Safe Harbor(%) 



Actual 

Proficiency 

Required 




Page 9 of 18 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0509-109-EPRU.pdf 



True Proficiency and Projecting Growth 



Previous studies that project school outcomes relative to AYP requirements have 
used a variety of methods to project the change for each school and each subgroup over 
time. While some studies have used a single growth projection for all schools and 
subgroups (e.g., average statewide change in proficiency over the previous year), others 
have simply continued each school’s observed growth over a previous period. 

This study employs a different method for projecting growth. Based on an 
empirical study of annual growth rates observed over three years (2002, 2003, and 2004) 
in Illinois (the only state for which sufficiently detailed data were available), this analysis 
has used quartiles of growth within starting bands of 10 percent proficiency to best match 
the current proficiency status of each subgroup with likely rates of change in proficiency 
over subsequent years. 

Our method is advantageous because it recognizes that different schools and 
subgroups start at different levels of proficiency. Because of these different starting 
points, subgroups must achieve different rates of growth to reach the 100 percent 
proficiency goal by 2014 and schools should reasonably be expected to progress at 
different rates. For example, growth expectations for schools starting out with 90 percent 
of students meeting the standards in 2004 cannot be expected to achieve the same level of 
annual growth as schools starting out with 10 percent of students at proficiency. Changes 
in observed proficiency should be expected to differ by school type in addition to starting 
proficiency. Elementary schools will likely not only progress at a different rate than 
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middle or high schools, but standard setting assumptions underlying proficiency 



designations differ across elementary, middle, and high schools as well. 

Studies that employ a common growth rate across all schools fail to recognize the 
important relationship between starting point, differential growth rates, and school type. 
On the other hand, studies that project each school’s rates of change based on unique 
observed rates over previous years run the risk of projecting extreme tails of the 
distribution of school proficiency changes (i.e., particularly high or low observed changes 
over short time periods) across longer time periods than they should be expected to 
sustain. Such methods disregard regression effects likely to mediate the variability in 
annual changes when considered over longer time periods. 

Projected rates of change in this study are conditional upon both starting point and 
school type. For each school type (elementary, middle and high school), the growth rates 
are projected uniquely by 10 percent bands according to the school’s starting point in 
2004 (0 to 10 percent students meeting standards, 11 to 20 percent of students meeting 
standards, etc.). 

The estimated impact of AYP is projected based on three possible scenarios 
(High, Medium, and Low Growth). The scenarios correspond to the observed annual 
growth rates at the 75 th , 50 th , and 25 th percentiles of schools within each starting band in 
the empirical study. For example, under the High Growth assumption, elementary 
schools that start out with between 10 percent and 20 percent proficient are projected to 
grow at an annual rate equivalent to the 75th percentile of annual growth observed for 
schools starting out between 10 percent and 20 percent proficiency in the three-year 
Illinois study. 
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The “true” proficiency of a school or subgroup cannot be observed, but the 



observed proficiency provides the best estimate for this true proficiency. As such, true 
proficiency “starting points” were estimated for each school and for each subgroup based 

o 

on the observed proficiency in 2003-2004. Projections under each scenario were made 
on the basis of changes in true proficiency relative to these starting estimates. 

The rates of change in proficiency by school type, subject area, and starting 
proficiency observed in Illinois from 2002-2004 are listed in Table 1. As is likely in any 
state, some schools evidenced declines in proficiency over these three years. For this 
study, however, no negative changes in proficiency were included; for cases in which 
observed proficiency did in fact decline, changes in proficiency were projected to be flat 
rather than negative. Table 2 includes the actual projections included as part of this 
study. 9 



Page 12 of 18 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0509-109-EPRU.pdf 





Table 1: Changes in Proficiency observed in Illinois, 2002-2004 



Elementary 



Initial Performance Decile (% Proficient) 



Growth 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90+ 

ELA Low 2.8 1.5 0.2 (0.2) (0.4) (0.8) (1.6) (2.7) (3.3) (4.2) 

Med 5.0 3.4 2.6 2.3 2.2 1.5 0.6 (0.7) (1.0) (2.3) 

High 8.3 6.0 5.9 5.5 5.0 4.2 2.5 1.2 0.5 (0.8) 

Math Low 7.2 3.3 2.7 2.4 1.9 1.4 1.0 (0.2 (0.8) (1.8) 

Med 12.2 5.8 6.1 6.3 5.7 5.0 3.4 1.9 0.8 (0.3) 

High 15.5 9.6 9.4 9.8 9.0 7.6 5.5 3.7 2.1 0.7 



Middle 

School 



Initial Performance Decile (% Proficient) 



Growth 


0-10 


10-20 


20-30 


30-40 


40-50 


50-60 


60-70 


70-80 


80-90 


90+ 


ELA Low 


9.7 


3.4 


3.2 


1.0 


- 


0.1 


(1.3) 


(2.5) 


(3.0) 


(3.4) 


Med 


9.7 


5.4 


4.9 


2.6 


2.1 


1.7 


0.7 


(0.9) 


(1.0) 


(1-4) 


High 


9.7 


9.2 


6.9 


5.1 


4.3 


4.3 


2.3 


0.8 


0.2 


0.0 



Math Low 


1.4 


(0.1) 


0.3 


(0-2) 


(0-6) 


(0.7) 


(1.4) 


(3.3) 


(2.8) 


(6.7) 


Med 


3.1 


2.6 


2.7 


2.7 


2.0 


1.2 


0.4 


(0-8) 


(0.8) 


(1-4) 


High 


5.8 


5.2 


5.3 


5.9 


4.4 


3.1 


2.8 


1.1 


0.8 


0.0 




High School Initial Performance Decile (% Proficient) 

Growth 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90+ 

ELA Low 1.0 (0.6) (0.5) (0.4) 0.3 (1.7) (2.6) (3.1) (2.2) (8.5) 

Med 2.5 0.2 1.3 1.3 1.8 (0.0) (1.1) (1.2) (0.6) (0.0) 

High 4.1 1.3 4.3 3.7 4.1 1.7 0.4 (0.2) (0.1) 0.1 

Math Low (0.2) (0.7) (0.6) (0.4) (1.4) (2.6) (3.2) (2.8) (0.5) 0.4 

Med 0.3 (0.0) 0.3 1.7 0.2 (0.9) (1.5) (1.4) (0.3) 1.0 

High 1.6 0.6 2.5 3.7 2.6 1.0 0.1 (0.3) 0.4 1.6 
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Table 2: Growth Projections Used in AYP Study 



Elementary 



Initial Performance Decile (% Proficient) 





Growth 


0-10 


10-20 


20-30 


30-40 


40-50 


50-60 


60-70 


70-80 


80-90 


90+ 


ELA 


Low 


2.8 


1.5 


0.2 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 




Med 


5.0 


3.4 


2.6 


2.3 


2.2 


1.5 


0.6 


0.0 


0.0 


0.0 



High 8.3 6.0 5.9 5.5 5.0 4.2 2.5 1.2 0.5 0.0 



Math 


Low 


7.2 


3.3 


2.7 


2.4 


1.9 


1.4 


1.0 


0.0 


0.0 


0.0 




Med 


12.2 


5.8 


6.1 


6.3 


5.7 


5.0 


3.4 


1.9 


0.8 


0.0 



High 15.5 9.6 9.4 9.8 9.0 7.6 5.5 3.7 2.1 0.7 



Middle 

School 



Initial Performance Decile (% Proficient) 





Growth 


0-10 


10-20 


20-30 


30-40 


40-50 


50-60 


60-70 


70-80 


80-90 


90+ 


ELA 


Low 


9.7 


3.4 


3.2 


1.0 


0.0 


0.1 


0.0 


0.0 


0.0 


0.0 




Med 


9.7 


5.4 


4.9 


2.6 


2.1 


1.7 


0.7 


0.0 


0.0 


0.0 




High 


9.7 


9.2 


6.9 


5.1 


4.3 


4.3 


2.3 


0.8 


0.2 


0.0 
















Math 


Low 


1.4 


0.0 


0.3 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 




Med 


3.1 


2.6 


2.7 


2.7 


2.0 


1.2 


0.4 


0.0 


0.0 


0.0 




High 


5.8 


5.2 


5.3 


5.9 


4.4 


3.1 


2.8 


1.1 


0.8 


0.0 



High School Initial Performance Decile (% Proficient) 



Growth 


0-10 


10-20 


20-30 


30-40 


40-50 


50-60 


60-70 


70-80 


80-90 


90+ 


ELA Low 


1.0 


0.0 


0.0 


0.0 


0.3 


0.0 


0.0 


0.0 


0.0 


0.0 


Med 


2.5 


0.2 


1.3 


1.3 


1.8 


0.0 


0.0 


0.0 


0.0 


0.0 


High 


4.1 


1.3 


4.3 


3.7 


4.1 


1.7 


0.4 


0.0 


0.0 


0.1 



Math Low 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 





Med 


0.3 


0.0 


0.3 


1.7 


0.2 


0.0 


0.0 


0.0 


0.0 


1.0 




High 


1.6 


0.6 


2.5 


3.7 


2.6 


1.0 


0.1 


0.0 


0.4 


1.6 



One last aspect of NCLB may change the requirements schools must meet to 
make AYP. An expansion of state testing programs built into NCLB requires that all 
states administer its assessments to students in grades three through eight and high school 
beginning in 2005-2006. Because some states are currently testing in only a subset of 
these grades (e.g., grades four, eight, and ten), this change will bring to these states an 
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increase of students participating in their testing programs. As such, the number of 
students within each subgroup will likely increase, possibly bringing some smaller 
subgroups into “numerical significance” for the first time. Estimates in this analysis take 
into account the expansion of state testing programs, and how more students being tested 
will affect the size and numerical significance of subgroups. Though we do account for 
increases in the number of grades tested, we do not account for any possible shifts in the 
relative demographics of school enrollments. 



AYP Failure Projections: Conservative Estimates 

Projections in this study were made for school-wide proficiency as well as 
proficiency of individual ethnic minority subgroups. NCLB actually requires 
measurement of all numerically significant subgroups - including not only ethnic 
minority students, but also students with limited English proficiency (LEP), students with 
disabilities, and socioeconomically disadvantaged students. In this study, only ethnic 
minority subgroups are assessed, as they are the only ones that are mutually exclusive or 
non-overlapping. In effect, schools are given a “free pass” for the requirements that other 
non-ethnic minority subgroups must meet under AYP. 

Projections are made for only mathematics and ELA. No projections are made for 
science, which will be phased in beginning in 2007, as starting proficiencies cannot yet 
be reliably determined in areas for which no standardized assessments are yet in place. 
The inclusion of Science as a required assessment area will make even more formidable 
the already challenging omnibus requirement for AYP success, in some cases increasing 
the number of participation and proficiency targets that a given school must meet by 50 
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percent. The current analysis does not include the addition of science, even though 



schools may eventually fail because of missed science targets. 

In other ways as well, the estimates detailed herein represent conservative 
projections of AYP success and failure. In addition to the aforementioned ways schools 
can fail AYP under NCLB, these projections exclude testing participation requirements. 
Under NCLB, schools and subgroups must not only meet proficiency targets but must 
also meet an assessment “participation” target (95 percent of students taking the state 
assessment). In this analysis, all schools and subgroups are assumed to have met the 
participation target, meaning that no schools are estimated to have failed AYP on that 
basis. 

This analysis has included only those requirements which can be reliably 
projected. As we have noted above, the accompanying estimates provide an upper bound 
for the true AYP success rates; in reality, schools could meet the requirements included 
in this analysis and still fail to make AYP by missing one of the targets (such as 
participation) excluded here. Even these upper bound estimates, however, project that 
school success relative to the AYP requirements as they currently stand will be quite low. 
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